Adaptive lead generation for marketing

ABSTRACT

Various examples are directed to systems and methods for adaptively generating leads. A marketing system may determine that a first lead score for a first lead is greater than a first lead score threshold and determine that a second lead score for a second lead is less than the first lead score threshold. The marketing system may generate a set of filtered leads including the first lead information from the first lead. The marketing system may determine a scrub rate that describes a portion of first execution cycle data having lead scores greater than the first lead score threshold and determine that the scrub rate is greater than an analysis window scrub rate by more than a scrub rate threshold. The marketing system may select a second lead score threshold that is lower than the first lead score threshold.

CLAIM OF PRIORITY

This patent application is a division of U.S. patent application Ser.No. 15/594,104, filed on May 12, 2017, which application claims thebenefit of priority, under 35 U.S.C. Section 119(e) to U.S. ProvisionalPatent Application Ser. No. 62/336,514, filed on May 13, 2016, which arehereby incorporated by reference herein in their entireties.

TECHNICAL FIELD

Examples described herein generally relate to systems and methods foraccurately and efficiently generating marketing leads.

BACKGROUND

Computer technology is used to generate marketing leads for companiesthat provide goods and/or services for sale.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. Some examples are illustrated by way of example, and notlimitation, in the figures of the accompanying drawings in which:

FIG. 1 is a diagram showing one example of an environment for adaptivelead generation.

FIG. 2 is a diagram showing another example of the environment of FIG. 1with additional components.

FIG. 3 is a diagram showing one example of the adaptive scoringcomponent.

FIG. 4 is a flowchart showing one example of a process flow 400 that maybe executed to receive and process lead score data 318.

FIG. 5 is a flowchart showing one example of a process flow 500 that maybe executed by the execution component of FIG. 3 to generate proposedchanges to the lead scorer and/or the lead filter.

FIG. 6 is a flowchart showing one example of a process flow 600 that maybe executed by the API of FIG. 3 to process new scoring and/or filteringparameters.

FIG. 7 is a block diagram showing one example of a software architecturefor a computing device.

FIG. 8 is a block diagram illustrating a computing device hardwarearchitecture, within which a set or sequence of instructions may beexecuted to cause the machine to perform examples of any one of themethodologies discussed herein.

DETAILED DESCRIPTION

In the following description, for purposes of explanation, numerousspecific details are set forth in order to provide a thoroughunderstanding of some examples. It will be evident, however, to oneskilled in the art that the present disclosure may be practiced withoutthese specific details.

Various examples are directed to systems and methods for adaptive leadgeneration. A lead includes data describing a potential customer for agood and/or service. Leads may be used to direct targeted advertising.For example, an advertiser may send advertising communications to apotential customer that are more extensive, personalized, and/orexpensive that what the advertiser would send to the general public. Alead may convert if the potential customer described by a lead purchasesa product or service from the advertiser or performs an action desiredby the advertiser. For example, a lead may be considered to convert ifthe potential customer fills out a survey or performs another similaraction requested by the advertiser.

A computerized marketing system is programmed to generate and filterleads. The marketing system may generate leads from various differentsources. For example, the marketing system, or other suitable system,may provide advertising content to one or more publishers. Publishersdistribute content to the public, for example, via the Internet. Apublisher incorporates ad content into the content that it provides tothe public. The advertising content may include a hyperlink or otherlink that is selectable by a potential customer to access a leadgenerator interface provided by the marketing system.

Through the lead generator interface, the marketing system may promptthe potential customer to provide information about him or herself,referred to as lead information. Lead information may includeinformation such as the potential customer's name, age, geographiclocation, etc. In some examples, lead information also includesinformation relevant to the advertised goods and/or service. Forexample, if the advertiser is an educational institution, the leadgeneration interface may prompt the potential customer to provide datasuch as, age, highest level of education achieved, how soon thepotential customer intends to begin schooling, etc. In another example,if the advertiser sells home security systems, the lead generationinterface may prompt the potential customer to provide data such as,whether the potential customer owns or rents a house, whether thepotential customer has experienced a break-in, etc. The marketing systemmay receive information from the potential customer and format theinformation into a lead.

The marketing system is also programmed to filter leads. For example,some leads are more likely to convert than others. The marketing systemmay train a model to correlate lead information to the likelihood that alead will convert. Any suitable model may be used. The marketing systemmay utilize the model to provide a lead score to the leads, where thelead score for a lead is a value indicating a probability that the leadwill convert (e.g., the likelihood that the potential customer describedby a lead will purchase a good and/or service from the advertiser). Themarketing system may generate filtered leads based on the assigned leadscore. In some examples, filtered leads include only leads having a leadscore indicating a probability that the leads will convert is greaterthan a lead score threshold value.

The marketing system may be programmed to adapt the lead scoring and/orfiltering process to changes in the quality of incoming leads. Leadquality may change for various reasons. In some examples, a singlepublisher may publish the same ad content through multiple web pages orother sources. For example, a publisher may produce a serious news webpage along with a less serious or tabloid-type news web page. Leads fromreaders of one web page over the other may be more likely to convert.Over time, the mix of leads provided by the publisher may change,causing a change to the quality of leads from the publisher.

The marketing system may be programmed to monitor a scrub rate of leadsfrom a particular publisher. If the scrub rate deviates from itshistorical value by more than a threshold amount, the marketing systemmay adapt the lead scoring and/or filtering process. Adapting the leadscoring process may include changing one or more scoring parametersaffecting the way that lead scores are determined. For example, scoringparameters may include coefficients of a predictive model used to assignlead scores to leads. Adapting the filtering process may includechanging one or more filtering parameters affecting the way that thescored leads are filtered. For example, a lead score threshold forfiltering may be a filtering parameter. In this way, the marketingsystem may adapt to changes in lead score quality. When a scoring and/orfiltering parameter is modified, subsequent leads may be scored and/orfiltered with the new scoring and/or filtering parameters.

FIG. 1 is a diagram showing one example of an environment 100 foradaptive lead generation. The environment includes a computerizedmarketing system 102, publisher systems 104A, 104B, an advertiser system106A and potential customers 108. The marketing system 102 may compriseone or more servers or other computing devices programmed to adaptivelygenerate leads, as described herein. The marketing system 102 maycomprise components 110, 112, 118, 120. Components 110, 112, 118, 120may be implemented by the marketing system 102 in any suitablecombination of hardware or software. Similarly, the publisher systems104A, 104B and advertiser system 106A may comprise one or more serversor other computing devices programmed to execute as described herein.Potential customers 108 may include people who interact with one or moreof the systems 102, 104A, 104B, 106A, as described herein. Potentialcustomers 108 may interact with one or more of the systems 102, 104A,104B, 106A utilizing any suitable computing device, for example, asdescribed in FIG. 2.

The marketing system 102 may, optionally, include an ad contentcomponent 110. The ad content component 110 may comprise one or moreprogrammed servers or other computing devices. In some examples, the adcontent component 110 may be executed on a common computing device withone or more of the other components 112, 114, 118, 120 of the marketingsystem 102. The ad content component 110 provides ad content 124 to oneor more of the publisher systems 104A, 104B. Although two publishersystems 104A, 104B are shown, the ad content component 110 may providead content 124 to any suitable number of publisher systems. The adcontent 124 may include information describing goods and/or servicesprovided by an advertiser or other entity. Also, in some examples, thead content 124 may include a Universal Resource Locator (URL) or othersuitable address for the lead interface 122, described in more detailbelow.

Publisher systems 104A, 104B provide publisher interfaces 126A, 126B,126C, 126D to the potential customers 108. Publisher interfaces 126A,126B, 126C, 126D may include content provided by the publisher systems126A, 126B (publisher content) and ad content that is or is derived fromad content 124. Publisher content and ad content included in thepublisher interfaces 126A, 126B, 126C, 126D may include, for example,text, images, audio files, video files, etc. Publisher interfaces 126A,126B, 126C, 126D may include any suitable interface or interfaces forproviding publisher content to the potential customers 108. For example,one or more of the publisher interfaces 126A, 126B, 126C, 126D mayinclude a website served to the potential customers. In some examples,one or more of the publisher interfaces 126A, 126B, 126C, 126D mayinclude an e-mail sent to the potential customers. Although fourpublisher interfaces 126A, 126B, 126C, 126D are shown in FIG. 1, anysuitable number of publisher interfaces may be included in theenvironment 100. In some examples, a single publisher system 104A mayserve more than one publisher interface 126A, 126B, 126C, 126D.

The potential customers 108 may receive one or more of the publisherinterfaces 126A, 126B, 126C, 126D. As described above, publisherinterfaces 126A, 126B, 126C, 126D may include a link to the leadinterface 122. A potential customer 108 with interest in the goodsand/or services described by the ad content in a publisher interface126A, 126B, 126C, 126D may select the link in the interface 126A, 126B,126C, 126D. The link may point to the lead interface 122, which may beserved by a lead generator component 114 of the marketing system 102.

The lead generator component 114 may comprise one or more programmedservers or other computing devices. In some examples, the lead generatorcomponent 114 may be executed on a common computing device with one ormore of the other components 110, 112, 118, 120 of the marketing system102. The lead generator component 114 may generate and serve the leadinterface 122 to one or more of the potential customers 108 (e.g., oneor more of the potential customers 108 who select the described link ina publisher interface 126A, 126B, 126C, 126D). The lead interface 122may prompt a potential customer to provide lead information about thepotential customer. The lead information may include information aboutthe potential customer including, for example, the potential customer'sname, age, e-mail address, mailing address, phone number, etc. Leadinformation may also include information specific to the product and/orservice indicated by the ad content. For example, where the product ishome-delivered food service, the lead interface 122 may prompt thepotential customer to provide the number of people in their household,the number of meals that their household eats at home in a week, etc. Inanother example where the product is a dry-cleaning service, the leadinterface 122 may prompt the potential customer to provide the number ofdry cleaned items that the potential customer's household wears in aweek, etc. The lead generator component 114 may be programmed to receiveand process lead data received from the potential customers 108 via thelead interface 122 into leads. As described above, a lead may includelead information describing a potential customer. The lead generatorcomponent 114 may process lead data into leads in any suitable format.

In some examples, the lead generator component 114 also categorizesleads. Any suitable categories may be used. For example, leads may becategorized by vertical, by advertiser, by product, by publisher, etc. Avertical or business area category for a lead may describe a category ofgoods and/or services that the potential customer may be interested inpurchasing. Example verticals include education (for potential customerswith interest in attending an educational institution), home security(for potential customers interested in purchasing a home security systemand/or service), insurance (for potential customers interested inpurchasing car, home, or other insurance), etc. An advertiser categoryfor a lead may describe the advertiser whose products are of interest tothe potential customer. For example, an advertiser may be a particulareducational institution, a particular home security system, etc. Aproduct category for a lead may describe a particular product (orservice) of interest to the potential customer. An example productwithin a home security vertical for a particular advertiser may be aparticular model of security system or a particular type of monitoringservices. A publisher category for a lead may describe the publisherfrom which the lead was generated. For example, if the potentialcustomer described by a lead accessed the lead interface 122 via a linkfrom the publisher system 104A, the publisher category of the resultinglead may describe the publisher implementing the publisher system 104A.In some examples, a category or categories for a lead may be embeddedinto the lead itself. For example, data describing the potentialcustomer may be supplemented with data describing one or more categoriesof the lead.

A lead scorer component 118 may apply a predictive model to leadsgenerated by the lead generator component 114 and assign to eachconsidered lead a lead score. The lead scorer component 118 may compriseone or more programmed servers or other computing devices. In someexamples, the lead scorer component 118 may be executed on a commoncomputing device with one or more of the other components 110, 112, 114,120 of the marketing system 102. Any suitable predictive model may beused including, for example, a decision tree or random forestcorrelation model, a linear regression model, a non-linear regressionmodel, an evolutionary model, a neural network model, a Bayesian model,etc. The lead scorer component 118 may apply the predictive model to alead to generate a lead score. The lead score is a value indicating aprobability that the lead will convert (e.g., the probability that thepotential customer described by the lead will purchase a good or servicefrom the advertiser). The lead score may be expressed on any suitablescale such as, for example, from 0 to 1, from 0 to 100, etc. The leadscorer component 118, in some examples, may execute in a cyclic manner.For example, the lead scorer component 118 may operate once a day, twicea day, once an hour, etc. During each operation, the lead scorercomponent 118 may act on leads received since the execution of itsprevious cycle. These may be referred to as cycle leads or executioncycle data. Execution cycle data may be received by the lead scorercomponent 118 continuously, intermittently, or in batches.

In some examples, the lead scorer component 118 also trains thepredictive model. For example, the lead scorer component 118 may receiveconversion data for a training set of leads, for example, from anadvertiser. The training set of leads may be previously-generated leadsthat were provided to the advertiser. The advertiser, or other suitableparty, may generate conversion data for the training set of leads bytallying whether the training leads were successfully converted (e.g.,whether the potential customer described by the lead purchased a good orservice from the advertiser). Accordingly, the lead scorer component 118may select predictive scoring parameters based on the training set andthe conversion data. Predictive scoring parameters may include, forexample, coefficients of one or more equations of the predictive model,

A lead filter component 120 may filter leads generated by the leadgenerator component 114 and scored by the lead scorer component 118. Thelead filter component 120 may comprise one or more programmed servers orother computing devices. In some examples, the lead filter component 120may be executed on a common computing device with one or more of theother components 110, 112, 114, 118 of the marketing system 102. In someexamples, the lead filter component 120 may apply a lead scorethreshold. Leads with a lead score exceeding the lead score thresholdmay be included in a set of filtered leads 130. Leads with a lead scoreless than or equal to the lead score threshold may not be included inthe filtered leads 130. Because a lead score describes the likelihoodthat a potential customer will convert (e.g., purchase the advertisedproduct and/or service), the filtered leads 130 may describe potentialcustomers that are most likely to convert. Filtered leads 130 may beprovided to an advertiser or other party that, for example, may use thefiltered leads 130 to direct additional marketing. The lead filtercomponent 120 may apply filter parameters such as, for example, the leadscore threshold. The filtering parameters may be changeable, asdescribed herein. When the lead filter component 120, for example,receives a new lead score threshold for leads (or for leads in certaincategories or combinations of categories), subsequent leads (orsubsequent leads in the indicated categories) may be filtered accordingto the new lead score threshold.

In some examples, the environment 100 also includes one or moreadvertiser systems 106A. Advertiser systems, such as the exampleadvertiser system 106A, may be implemented by an advertiser (e.g., abusiness entity that is selling goods and/or services or arepresentative of the business entity that is selling goods and/orservices). The advertiser system 106A may provide to the potentialcustomers an advertiser interface 128 that provides content related tothe goods and/or services for sale. For example, when the advertiser isan educational institution, the advertiser interface 128 may be awebsite for the advertiser. The advertiser interface 128 may comprise alink to the lead interface 122 that may allow potential customers 108who receive the advertiser interface 128 to select the link and proceedto the lead interface 122 as described herein.

FIG. 2 is a diagram showing another example of the environment 100 ofFIG. 1 with additional components. FIG. 2 shows the marketing system102, publisher systems 104A, 104B, and advertiser system 106A describedabove. FIG. 2 also shows an additional advertiser system 106B. Forexample, the marketing system 102 may be programmed to execute adaptivelead generation for more multiple advertisers at the same time. It willbe appreciated that any suitable number of the systems 104A, 104B, 106A,106B, 102, etc. may be included in the environment 100.

FIG. 2 also shows potential customers 108A, 108B, 108C utilizingcustomer computing devices 109A, 109B, 109C. Although three customers108A, 108B, 108C and three customer computing devices 109A, 109B, 109Care shown in FIG. 2, any suitable number of potential customers 108A,108B, 108C may receive publisher interfaces 126A, 126B, 126C, 126Dand/or advertiser interface 128. Similarly, any suitable number ofpotential customers 108A, 108B, 108C may interact with the leadinterface 122.

Potential customers 108A, 108B, 108C may receive and interact withpublisher interfaces 126A, 126B, 126C, 126D, advertiser interface 128,and/or lead interface 122 utilizing customer computing devices 109A,109B, 109C. Customer computing devices 109A, 109B, 109C may include anycomputing device suitable for receiving and/or interacting with a userinterface. For example, customer computing device 109A may be a tabletcomputing device and/or mobile phone. Customer computing device 109B maybe a laptop computer. Customer computing device 109C may be a desktopcomputer. Customer computing devices 109A, 109B, 109C, however, mayinclude any other suitable computing device.

The various components of the environment 100 may be in communicationwith one another via a network 132. The network 132 may be or compriseany suitable network element operated according to any suitable networkprotocol. For example, one or more portions of network 132 may be an adhoc network, an intranet, an extranet, a virtual private network (VPN),a local area network (LAN), a wireless LAN (WLAN), a wide area network(WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), aportion of the Internet, a portion of the Public Switched TelephoneNetwork (PSTN), a cellular telephone network, a wireless network, aWi-Fi network, a WiMax network, another type of network, or acombination of two or more such networks.

FIG. 3 is a diagram showing one example of the adaptive scoringcomponent 112. The adaptive scoring component 112 may comprise a datapreparation component 302, an execution component 303, and an initialsetup component 317. The initial setup component 317 may be configuredto initialize the operation of the adaptive scoring component 112, forexample, as described herein. In some examples, the adaptive scoringcomponent 112 may be implemented utilizing the Apache Spark clustercomputing framework.

The data preparation component 302 may receive lead score data 318. Leadscore data 318 may describe received leads and lead scores assigned tothe leads by the lead scorer component 118. In some examples, the leadscore data 318 is real time data. For example, the lead score data 318may include lead scores generated by the lead scorer component 118 asthe lead scores are generated. Lead scores are calculated based onanswers provided by the potential customers to questions posed via thelead interface 122. In some examples, each lead (e.g., each potentialcustomer) may receive an individual lead score. In some examples, thelead score data may include an estimated lead score, a current cut-offthreshold to control if the lead/consumer has a sufficient likelihood toconvert and submit to an advertiser. The data preparation component 302may receive the lead score data 318 and format the data for analysis andaction by the execution component 303.

In addition to lead score data, the adaptive scoring component 112 may,in some examples, receive and/or derive traffic data. Traffic data mayinclude traffic statistics describing incoming leads such as, forexample, lead volume, a volume or rate of leads scrubbed (e.g., omittedfrom the filtered leads 130), etc. In some examples, traffic data may bebroken out by category or combination of categories. For example, leadscore data 318 may include a description of the scrub rate for leads ina particular combination of date, vertical category, product category,advertiser category, and publisher category.

In some examples, the data preparation component 302 includes a dataextraction and transformation component 314 and a data consolidatorcomponent 316. The data extraction and transformation component 314 mayperform various formatting, deduplication, transformation and/oraggregation operations on the lead score data 318. The data consolidatorcomponent 316 may generate records of aggregated lead score data, whichmay be stored at a database 320. Additional details of the operation ofthe data preparation component 302 are provided herein, for example,with respect to the process flow 400 of FIG. 4.

The execution component 303 may comprise an adjustment evaluatorcomponent 304. The execution component 303 may also comprise a guardrails enforcer 306 and/or an adaptive model executor component 308. Theadjustment evaluator component 304 may receive lead score data 318 viathe data preparation component 302. Based on the lead score data 318,the adjustment evaluator may determine an adjustment to the lead scorercomponent 118 and/or the lead filter component 120. An adjustment to thelead filter component 120 may be, for example, a change to the leadscore threshold value applied by the lead filter component 120 or otherfiltering parameter. The guard rails enforcer component 306 may applyone or more limitation rules to the lead filter component or lead scorercomponent adjustment determined by the model scrub adjustment evaluatorcomponent 304. The optional model executor component 308 may derive achange to the predictive model used by the lead scorer component 118 toassign lead scores to leads. For example, the model executor component308 may re-train the predictive model utilized by the lead scorercomponent 118, for example, based on newly received leads. In someexamples, the model executor component 308 may automatically regeneratean entire predictive model utilized by the lead scorer component 118,for example, based on new and ever-changing input. Changes to the leadfilter component 120 and/or the lead scorer component 118 may becommunicated to one or both of those components 118, 120 via a leadscorer/lead filter application program interface (API) 315.

In some examples, the adaptive scoring component comprises a recordsintegration component 310. The records integration component 310 may beutilized to document changes made to the lead scorer component 118and/or the lead filter component 120. For example, when the executioncomponent 303 pushes a change to either the lead scorer component 118 orthe lead filter component 120 (e.g., via the API 315), it may also beprogrammed to initiate a records task ticket, which is provided to therecords integration component 310. The records integration component 310may be implemented with any suitable technology. In some examples, therecords integration component 310 includes and/or interfaces with aproject management tool, such as the JIRA product available fromAtlassian.

The records task ticket may include data describing the proposed changeto the lead scorer component 118 or lead filter component 120. Forexample, a records task ticket may include values for one or morescoring and/or filtering parameters both before and after the proposedchange. For example, if the proposed change includes a modification tothe lead score threshold applied by the lead filter 120 (e.g., afiltering parameter), the records task ticket may describe the previouslead score threshold and the proposed new lead score threshold. If thechange includes a modification to the lead scorer component 118, therecords task ticket may include a description of the model applied bythe lead scorer component 118 both before and after the proposed change(e.g., one or more scoring parameters). The description of the model mayinclude, for example, model coefficients, offsets, or any other modeldescription. In some examples, a records task ticket may also include adescription of the scrub rate or other metric describing the operationof the lead scorer component 118 or lead filtering component 120 priorto the change.

Upon receiving a records task ticket, the records integration component310 may monitor the scrub rate or other indicator of the operation ofthe lead scorer component 118 and/or lead filtering component 120. Therecords integration component 310 may generate a records entry includingsome or all of the data from the records task ticket and the scrub rateor other indicator of the operation of the lead scorer component 118both before and after the proposed change in implemented. The recordsentry may be stored at any suitable database including, for example, thedatabase 320.

The alerting component 312 may also receive an alert ticket from theexecution component 303 when a change is proposed to the lead scorercomponent 118 and/or the lead filter component 120. The alert ticket maydescribe a proposed change to the lead scorer component and/or leadfilter component 120 (e.g., a change to a scoring and/or filterparameter). In some examples, an alert ticket may include data similarto the data described above as being included in the records taskticket. In response to the alert ticket, the alerting component 312 maygenerate and send a message to one or more administrators of themarketing system 102. The message may be sent, for example, by e-mail,text (e.g., short message service SMS), or any other suitable medium.The message may include some or all of the data described above aspotentially included in a records entry generated by the recordsintegration component 310. In some examples, the alerting component 312may send a message describing a proposed change to the lead scorercomponent 118 or the lead filter component 120 before the change isimplemented. This may enable the administrator receiving the message tomanually intercede to cancel or change the modification if desirable.For example, the execution component 303 may be programmed to send thealert ticket to the alerting component 312 and then wait a thresholdtime (e.g., 30 minutes) before executing the change to the lead scorercomponent 118 and/or lead filter component 120 described by the alertticket.

FIG. 4 is a flowchart showing one example of a process flow 400 that maybe executed to receive and process lead score data 318. The process flow400 may begin at action 402. The process flow 400 may begin at anysuitable interval or time. In some examples, the process flow 400 mayexecute when real-time lead score data 318 is received (e.g., lead scoredata 318 that was generated by the lead scorer component 118 within athreshold time of receipt and/or lead score data 318 that has not yetbeen processed by the data preparation component 302). Also, in someexamples, the process flow 400 may execute continuously.

At action 404, the initial setup component 317 may determine and/orreceive a vertical or business domain category of the lead score data318 to be processed. For example, the lead score data 318 may beformatted a setup file (or other suitable data structure). The initialsetup component 317 may examine the setup file to determine the verticalcategory of the leads therein. In addition, the data preparationcomponent 302 may determine an advertiser list, a product or productsrepresented by the lead score data 318 and publisher list, for example,from the setup file.

At action 406, the data preparation component 302 (e.g., the dataextraction and transformation component 314 thereof) may extract theversioned scoring and/or filtering parameters. For example, the dataextraction and transformation component 314 may query the lead scorercomponent 118 to receive the scoring and/or filtering parameters. Insome examples, the query is directed to a software versioning and/orrevision control server, which may execute any suitable versioningand/or revision control tool such as, Subversion® available fromApache™.

At action 408, the data preparation component 302 (e.g., the dataextraction and transformation component 314 thereof) may determine ifreal time lead score data 318 is available. For example, if real timelead score data is not available, it may indicate that the lead scorercomponent 118 and/or lead filter component 120 is not operational (e.g.,not generating lead score data 318). If real time lead score data 318 isnot available, the data extraction and transformation component 314 mayinitiate a message to an administrator of the marketing system 102. Themessage may be sent by any suitable medium including, for example,e-mail, SMS message, etc. The message may indicate that there is no newlead scoring data 318 received from the lead scorer component 118 and/orthe lead filtering component 120. The administrator, for example, mayintervene to determine why the lead scorer component 118 and/or leadfilter component 120 is not operating and, optionally, to correct theissue. The process flow 400 may subsequently stop at action 412.

If real time lead score data 318 is received, the data preparationcomponent 302 (e.g., the data extraction and transformation component314 thereof) may receive an indicator of an analysis window at action414. The analysis window may describe the time period over which theperformance of the lead scorer component 118 and/or lead filtercomponent will be evaluated. For example, the execution component maycompare a current scrub rate taken over an observation time period(e.g., a day, an hour) to a scrub rate for leads received over theanalysis window (e.g., a week, 15 days, 30 days, a quarter, etc.). Ataction 416, the data preparation component 302 (e.g., the dataextraction and transformation component 314 thereof) may receive anindicator of execution scope for the execution component 303. Theexecution scope may describe a range of lead categories that theexecution component 303 will analyze. For example, the execution scopemay comprise one or a set of advertiser categories, one or a set ofproduct categories, etc. The analysis window data and execution scopemay be received in any suitable manner. For example, an administrator ofthe marketing system 102 may set the analysis window and execution scopemanually.

At action 418, the data preparation component 302 (e.g., the dataextraction and transformation component 314 thereof) may perform dataanalysis on the lead score data. For example, the lead score data 318may be received in any suitable format including, for example, astructured or unstructured format. Data analysis may include extractinglead scores from the received data 318, transforming the received data318 to a structured format, such as a spreadsheet or eXtensible MarkupLanguage (XML), and loading the structured lead score data, for example,to the database 320 (e.g., via the data consolidator component 316).Data analysis, in some examples, also includes deduplication. In someexamples, lead score data 318 extracted, transformed, and/or loaded ataction 418 may include only the lead score data 318 describing leadsreceived during the analysis window.

At action 420, the data preparation component 302 (e.g., the dataextraction and transformation component 314 thereof) may receive and/orgenerate analysis window statistics describing the received lead scoredata over the analysis window. Example statistics that may be generatedinclude an average lead score for leads over the analysis period, ann-day moving average of lead scores for leads received during theanalysis window, a standard deviation of lead scores for leads receivedover the analysis period, etc. At action 422, the data preparationcomponent 302 (e.g., the data extraction and transformation component314 thereof) may receive and/or generate observation period statistics.Observation period statistics may be similar to the analysis windowstatistics, but may be taken over the observation period. As describedherein, the execution component 303 may compare observation periodstatistics to analysis window statistics to propose changes to the leadscorer component 118 and/or lead filter component 120. The statisticsdescribed at actions 418 and 420, in some examples, are received and/orextracted from the lead score data 318. Also, in some examples, some orall of the statistics of actions 418 and 420 are derived from the leadscore data 318 and/or from other sources. Also, in some examples, thedata preparation component 302 may receive the statistics described atactions 418 and 420 from another source (e.g., the records integrationcomponent 310).

At action 424, the data preparation component 302 (e.g., the dataconsolidator component 316 thereof) may consolidate the structured leaddata generated at action 418 and the statistics received and/orgenerated at actions 420, 422 to generate a master dataset. The masterdataset may be in any suitable format including, for example, XML, aspreadsheet, etc. The master dataset, in some examples, may be stored atthe database 320. At action 426, the data preparation component 302 mayprovide the consolidated data to the Execution Component 303.

FIG. 5 is a flowchart showing one example of a process flow 500 that maybe executed by the execution component 303 of FIG. 3 to generateproposed changes to the lead scorer 118 and/or the lead filter 120. Theprocess flow 500 is divided into two portions 501, 503. Portion 501includes actions that may be executed by the adjustment evaluatorcomponent 304. Portion 503 includes actions that may be executed by theguard rail enforcer component 306. For example, as described above, theadjustment evaluator component 304 may generate proposed changes toscoring and/or filtering parameters. The guard-rails enforcer component306 may apply limitation rules to limit the severity of changes made bythe adjustment evaluator component 304. At action 502, the executioncomponent 303 may receive input data from the data preparation component302. The input data, for example, may be the structured master datasetdescribed above. For example, the input data may include lead scoresand/or statistics from the analysis window and/or the observationperiod.

At action 504, the adjustment evaluator 304 may receive parameters foradaptive model updates. The parameters may include, for example, thefiltering parameters and/or scoring parameters that are currently beingapplied by the lead scorer component 118 and/or the lead filtercomponent 120.

At action 506, the adjustment evaluator component 304 may receiveadjustment rules. Adjustment rules may define when the adjustmentevaluator component 304 proposes a change to a scoring parameter and/ora filtering parameter. Any suitable adjustment rule may be utilized. Oneexample adjustment rule may require that the mean scrub rate implementedby the lead filter component 120 over the observation period be withinplus-or-minus two standard deviations of the mean scrub rate over theanalysis window. Another example adjustment rule may set a maximumand/or minimum scrub rate over the observation period. In some examples,adjustment rules may be specific to a particular set of lead categories.Leads belonging to a particular set of lead categories may be referredto as a domain.

In some examples, one or more adjustment rules may call formodifications to a filtering/scoring parameter and/or the minimumlead-score cut-off requirement upon detection of a change in the qualityof leads over time (e.g., from one execution cycle to another.) Forexample, lead quality during an observation period may be compared tolead quality during the analysis window. Lead quality may be measured inany suitable manner. For example, lead quality may be reflected in thescrub rate. If the scrub rate for leads received during the observationperiod is different than the scrub rate for leads received analysiswindow by more than a scrub rate threshold, it may indicate that leadquality has changed. In some examples, lead quality may be measured bycomparing lead determinative variables to lead scores. A leaddeterminative variable may be a variable describing a potential customerthat tends to suggest that the potential customer is either more likelythan average or less likely than average to convert. For example, in avertical category of car dealers, a potential customer without a driverslicense may be relatively less likely to convert (e.g., buy a car) thansimilarly situated potential customers that have drivers licenses. Alead determinative variable may conflict with a lead score if the leaddeterminative variable and the lead score differ. If more than athreshold number of leads in an execution cycle have lead determinativevariables differing from the determined lead score, it may indicate adrop in lead quality.

At action 508, the adjustment evaluator component 304 may evaluate theinput data in view of the adjustment rules and identify candidatescoring and/or filtering parameters for change. Changes may be proposedfor any suitable scope of leads. In some examples, changes may beproposed for leads assigned to categories similar to those of the leadsreceived by the data preparation component 302. For example, theadjustment evaluator component 304 may determine if any of theadjustment rules are satisfied by a unique combination of advertiser,vertical, product and publisher from the lead score data. Scrub rate maybe estimated for each unique combination of advertiser, vertical,product and publisher from lead score data. Scrub rate may be the ratioof number of customers whose lead score who are removed from marketingsystem 102 (because their lead score is below the current predefinedmodel cut-off/threshold) to the total number of customers in theMarketing System. If so, the adjustment evaluator component 304 maygenerate a proposed modification to the scoring parameters and/or thefiltering parameters. For example, if the scrub rate for the observationperiod is too high, the adjustment evaluator component 304 may propose areduction to the lead score threshold applied by the lead filteringcomponent 120.

Changes to scoring and/or filtering parameters may be implemented in anysuitable manner. For example, if the adjustment evaluator 304 determinesthat a change to the scrub rate is desired, it may analyze leadsreceived during the observation period and select a new lead scorethreshold that would drive the scrub rate to the desired value for theleads received during the observation period. This may be set as the newlead score threshold for the lead filter 120.

At action 510, the adjustment evaluator component 304 may determinewhether all of the adjustment rules have been evaluated. If not, theadjustment evaluator component 304 may return to action 508 and evaluateone or more additional adjustment rules. If all adjustment rules areevaluated, the adjustment evaluator component 304 may determine, ataction 512, whether any changes to filtering parameters and/or scoringparameters have been proposed. If not, the adjustment evaluatorcomponent 304 may, at action 534, send a message to one or moreadministrators indicating that no change to the filtering and/or scoringparameters is proposed. Upon initiating the message, the adjustmentevaluator 304 may stop its processing at action 536.

If a change to a filtering and/or scoring parameter is proposed, theadjustment evaluator 304 may, at action 514, inventory parameter changecandidates including the magnitude of the proposed changes. At action516, the adjustment evaluator component may log change candidatesincluding the recommended new scrub level. In some examples,inventorying parameter change candidates may include recording theproposed parameter changes as well as, for example, statisticsdescribing the lead scores that lead to the proposed changes, new andold scrub thresholds, etc. A generated inventory of parameter changesmay be provided, for example, to the records integration component 310for storage and/or distribution. A log of the proposed parameter changesmay include a less data than an inventory and, for example, may beprovided to the alerting component 312 for distribution toadministrative users. In some examples, one or both of actions 514 and516 may be omitted and/or combined into a single action.

At action 518, the guard rails enforcer component 306 may receivelimitation rules, sometimes referred to as guard rails or guard railrules. The guard rails enforcer component 306 may enforce thelimitations rules at action 520. Like adjustment rules, limitationrules, in some examples, are specific to a particular domain. Anysuitable limitation rules may be used. In some examples, a limitationrule may set minimum or maximum lead score thresholds. In anotherexample, a limitation rule may limit the number of positive and/ornegative changes that can be made to the lead score threshold over agiven time period. For example, if the adjustment evaluator 304 proposesmore than three increases to the lead score threshold in a one monthperiod, the guard rails enforcer component 306 may disallow the fourthand subsequent proposed increases during the one month period.

At action 522, the guard rails enforcer component 306 may determine ifall limitation rules have been enforced. If not, the guard railsenforcer component 306 may enforce additional limitation rules at action520. If all limitation rules have been enforced, the guard railsenforcer component 306 may consolidate all changes remaining afterapplication of the limitation rules at action 524. At action 526, theguard rails enforcer may call the lead scoring component 118 and/or thelead filter component 120 to receive current filtering and/or scoringparameters. Filtering and/or scoring parameters, in some examples, arerepresented utilizing Predictive Model Markup Language (PMML), thoughany other suitable syntax may be used. The guard rails enforcercomponent 306 may query the lead scorer component 118 and/or lead filtercomponent 120 via the API 315.

At action 528, the guard rails enforcer 306 may generate a new set oflead scoring and/or filtering parameters. For example, the guard railsenforcer 306 may modify the PMML or other model representation receivedfrom the lead scorer component 118 and/or lead filter component 120. Ataction 530, the guard rails enforcer 306 may perform version control onthe new scoring and/or filtering parameters. Version control mayinclude, for example, assigning a new version number to the parametersas modified. The guard rails enforcer 306, in some examples, may manageversion control in conjunction with a versioning and/or revision controlserver, as described herein. At action 530, the guard rails enforcercomponent 306 may call the API 315 and provide the new and/or updatedparameters to the lead scorer component 118 and/or the lead filtercomponent 120.

FIG. 6 is a flowchart showing one example of a process flow 600 that maybe executed by the API 315 to process new scoring and/or filteringparameters. At action 602, the API 315 may receive a call (e.g., fromthe execution component) requesting that new scoring filteringparameters and/or new lead score cut-off threshold to be deployed. Insome examples, the call may include a PMML or other representation ofthe model utilized by the lead scorer component 118 to score leadsand/or by the lead filter component 120 to filter leads. The PMML orother model representation, for example, may reflect modifications toscoring parameters and/or filtering parameters. In some examples,whether the call includes a full representation of a model or simplyparameters to be changed or updated, the call may refer to a modelexecuted by the lead scorer component 118 and/or the lead filtercomponent 120.

At action 604, the API 315 may deploy a new model to the lead scorercomponent 118 and/or the lead filter component 120. At action 606, theAPI 315 may call the records integration component 310, which may recordthe parameter changes and alert administrators, for example, asdescribed herein. At action 608, the API 315 may, if necessary, add tothe call made to the records integration component 310 an indication ofbefore and after values for the scoring and/or filtering parameters tobe changed. For example, when the lead score threshold is changed, thecall to the records integration component 310 may include an indicationof the lead score threshold both before and after the change. At action610, the API 315 may, optionally, send a message to one or moreadministrators indicating a successful change to the scoring and/orfiltering parameter. The process flow 600, in some examples, mayterminate at action 612.

FIG. 7 is a block diagram 700 showing one example of a softwarearchitecture 702 for a computing device. The architecture 702 may beused in conjunction with various hardware architectures, for example, asdescribed herein. FIG. 7 is merely a non-limiting example of a softwarearchitecture and many other architectures may be implemented tofacilitate the functionality described herein. A representative hardwarelayer 704 is illustrated and may represent, for example, any of theabove referenced computing devices.

The representative hardware layer 704 comprises one or more processingunits 706A having associated executable instructions 708. Executableinstructions 708 represent the executable instructions of the softwarearchitecture 702, including implementation of the methods, components,and so forth of FIGS. 1-6. Hardware layer 704 also includes memoryand/or storage components 710, which also have executable instructions708. Hardware layer 704 may also comprise other hardware as indicated by712 which represents any other hardware of the hardware layer 704, suchas the other hardware illustrated as part of hardware architecture 800of FIG. 8 below.

In the example architecture of FIG. 7, the software 702 may beconceptualized as a stack of layers where each layer provides particularfunctionality. For example, the software 702 may include layers such asan operating system 714, libraries 716, frameworks/middleware 718,applications 720 and presentation layer 744. Operationally, theapplications 720 and/or other components within the layers may invokeapplication programming interface (API) calls 724 through the softwarestack and receive a response, returned values, and so forth illustratedas messages 726 in response to the API calls 724. The layers illustratedare representative in nature and not all software architectures have alllayers. For example, some mobile or special purpose operating systemsmay not provide a frameworks/middleware layer 718, while others mayprovide such a layer. Other software architectures may includeadditional or different layers.

The operating system 714 may manage hardware resources and providecommon services. The operating system 714 may include, for example, akernel 728, services 730, and drivers 732. The kernel 728 may act as anabstraction layer between the hardware and the other software layers.For example, the kernel 728 may be responsible for memory management,processor management (e.g., scheduling), component management,networking, security settings, and so on. The services 730 may provideother common services for the other software layers. The drivers 732 maybe responsible for controlling or interfacing with the underlyinghardware. For instance, the drivers 732 may include display drivers,camera drivers, Bluetooth® drivers, flash memory drivers, serialcommunication drivers (e.g., Universal Serial Bus (USB) drivers), Wi-Fi®drivers, NFC drivers, audio drivers, power management drivers, and soforth depending on the hardware configuration.

The libraries 716 may provide a common infrastructure that may beutilized by the applications 720 and/or other components and/or layers.The libraries 716 typically provide functionality that allows othersoftware modules to perform tasks in an easier fashion than to interfacedirectly with the underlying operating system 714 functionality (e.g.,kernel 728, services 730 and/or drivers 732). The libraries 716 mayinclude system 734 libraries (e.g., C standard library) that may providefunctions such as memory allocation functions, string manipulationfunctions, mathematic functions, and the like. In addition, thelibraries 716 may include API libraries 736 such as media libraries(e.g., libraries to support presentation and manipulation of variousmedia format such as MPEG4, H.264, MP3, AAC, AMR, JPG, PNG), graphicslibraries (e.g., an OpenGL framework that may be used to render 2D and9D in a graphic content on a display), database libraries (e.g., SQLitethat may provide various relational database functions), web libraries(e.g., WebKit that may provide web browsing functionality), and thelike. The libraries 716 may also include a wide variety of otherlibraries 738 to provide many other APIs to the applications 720 andother software components/modules.

The frameworks 718 (also sometimes referred to as middleware) mayprovide a higher-level common infrastructure that may be utilized by theapplications 720 and/or other software components/modules. For example,the frameworks 718 may provide various graphic user interface (GUI)functions, high-level resource management, high-level location services,and so forth. The frameworks 718 may provide a broad spectrum of otherAPIs that may be utilized by the applications 720 and/or other softwarecomponents/modules, some of which may be specific to a particularoperating system or platform.

The applications 720 includes built-in applications 740 and/or thirdparty applications 742. Examples of representative built-in applications740 may include, but are not limited to, a contacts application, abrowser application, a book reader application, a location application,a media application, a messaging application, and/or a game application.Third party applications 742 may include any of the built-inapplications as well as a broad assortment of other applications. In aspecific example, the third-party application 742 (e.g., an applicationdeveloped using the Android™ or iOS™ software development kit (SDK) byan entity other than the vendor of the particular platform) may bemobile software running on a mobile operating system such as iOS™,Android™, Windows® Phone, or other mobile operating systems. In thisexample, the third-party application 742 may invoke the API calls 724provided by the mobile operating system such as operating system 714 tofacilitate functionality described herein.

The applications 720 may utilize built in operating system functions(e.g., kernel 728, services 730 and/or drivers 732), libraries (e.g.,system 734, APIs 736, and other libraries 738), frameworks/middleware718 to create user interfaces to interact with users of the system.Alternatively, or additionally, in some systems interactions with a usermay occur through a presentation layer, such as presentation layer 744.In these systems, the application/module “logic” may be separated fromthe aspects of the application/module that interact with a user.

Some software architectures utilize virtual machines. In the example ofFIG. 7, this is illustrated by virtual machine 748. A virtual machinecreates a software environment where applications/modules may execute asif they were executing on a hardware computing device. A virtual machineis hosted by a host operating system (operating system 714) andtypically, although not always, has a virtual machine monitor 746, whichmanages the operation of the virtual machine as well as the interfacewith the host operating system (i.e., operating system 714). A softwarearchitecture executes within the virtual machine such as an operatingsystem 750, libraries 752, frameworks/middleware 754, applications 756and/or presentation layer 758. These layers of software architectureexecuting within the virtual machine 748 may be the same ascorresponding layers previously described or may be different.

FIG. 8 is a block diagram illustrating a computing device hardwarearchitecture 800, within which a set or sequence of instructions may beexecuted to cause the machine to perform examples of any one of themethodologies discussed herein. For example, the architecture 800 mayexecute the software architecture 702 described with respect to FIG. 7.The architecture 800 may operate as a standalone device or may beconnected (e.g., networked) to other machines. In a networkeddeployment, the architecture 800 may operate in the capacity of either aserver or a client machine in server-client network environments, or itmay act as a peer machine in peer-to-peer (or distributed) networkenvironments. The architecture 800 may be implemented in a personalcomputer (PC), a tablet PC, a hybrid tablet, a set-top box (STB), apersonal digital assistant (PDA), a mobile telephone, a web appliance, anetwork router, switch or bridge, or any machine capable of executinginstructions (sequential or otherwise) that specify actions to be takenby that machine.

Example architecture 800 includes a processor unit 802 comprising atleast one processor (e.g., a central processing unit (CPU), a graphicsprocessing unit (GPU) or both, processor cores, compute nodes, etc.).The architecture 800 may further comprise a main memory 804 and a staticmemory 806, which communicate with each other via a link 808 (e.g.,bus). The architecture 800 may further include a video display unit 810,an alphanumeric input device 812 (e.g., a keyboard), and a userinterface (UI) navigation device 814 (e.g., a mouse). In some examples,the video display unit 810, input device 812 and UI navigation device814 are incorporated into a touch screen display. The architecture 800may additionally include a storage device 816 (e.g., a drive unit), asignal generation device 818 (e.g., a speaker), a network interfacedevice 820, and one or more sensors (not shown), such as a globalpositioning system (GPS) sensor, compass, accelerometer, or othersensor.

The storage device 816 includes a machine-readable medium 822 on whichis stored one or more sets of data structures and instructions 824(e.g., software) embodying or utilized by any one or more of themethodologies or functions described herein. The instructions 824 mayalso reside, completely or at least partially, within the main memory804, static memory 806, and/or within the processor 802 during executionthereof by the architecture 800, with the main memory 804, static memory806, and the processor 802 also constituting machine-readable media.Instructions stored at the machine-readable medium 822 may include, forexample, instructions for implementing the software architecture 702,instructions for executing any of the features described herein, etc.

While the machine-readable medium 822 is illustrated in an example to bea single medium, the term “machine-readable medium” may include a singlemedium or multiple media (e.g., a centralized or distributed database,and/or associated caches and servers) that store the one or moreinstructions 824. The term “machine-readable medium” shall also be takento include any tangible medium that is capable of storing, encoding orcarrying instructions for execution by the machine and that cause themachine to perform any one or more of the methodologies of the presentdisclosure or that is capable of storing, encoding or carrying datastructures utilized by or associated with such instructions. The term“machine-readable medium” shall accordingly be taken to include, but notbe limited to, solid-state memories, and optical and magnetic media.Specific examples of machine-readable media include non-volatile memory,including, but not limited to, by way of example, semiconductor memorydevices (e.g., electrically programmable read-only memory (EPROM),electrically erasable programmable read-only memory (EEPROM)) and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

The instructions 824 may further be transmitted or received over acommunications network 826 using a transmission medium via the networkinterface device 820 utilizing any one of a number of well-knowntransfer protocols (e.g., HTTP). Examples of communication networksinclude a local area network (LAN), a wide area network (WAN), theInternet, mobile telephone networks, plain old telephone (POTS)networks, and wireless data networks (e.g., Wi-Fi, 3G, and 6G LTE/LTE-Aor WiMAX networks). The term “transmission medium” shall be taken toinclude any intangible medium that is capable of storing, encoding, orcarrying instructions for execution by the machine, and includes digitalor analog communications signals or other intangible medium tofacilitate communication of such software.

Examples, as described herein, may include, or may operate on, logic ora number of components, engines, or modules, circuits, which for thesake of consistency are termed circuits, although it will be understoodthat these terms may be used interchangeably. Circuits may be hardware,software, or firmware communicatively coupled to one or more processorsin order to carry out the operations described herein. Circuits may behardware circuits, and as such circuits may be considered tangibleentities capable of performing specified operations and may beconfigured or arranged in a certain manner. In an example, circuits maybe arranged (e.g., internally or with respect to external entities suchas other circuits) in a specified manner as a circuit. In an example,the whole or part of one or more computing platforms (e.g., astandalone, client or server computing platform) or one or more hardwareprocessors may be configured by firmware or software (e.g.,instructions, an application portion, or an application) as a circuitthat operates to perform specified operations. In an example, thesoftware may reside on a machine-readable medium. In an example, thesoftware, when executed by the underlying hardware of the circuit,causes the hardware to perform the specified operations. Accordingly,the term hardware circuit is understood to encompass a tangible entity,be that an entity that is physically constructed, specificallyconfigured (e.g., hardwired), or temporarily (e.g., transitorily)configured (e.g., programmed) to operate in a specified manner or toperform part or all of any operation described herein.

Considering examples in which circuits are temporarily configured, eachof the circuits need not be instantiated at any one moment in time. Forexample, where the circuits comprise a general-purpose hardwareprocessor configured using software; the general-purpose hardwareprocessor may be configured as respective different circuits atdifferent times. Software may accordingly configure a hardwareprocessor, for example, to constitute a particular circuit at oneinstance of time and to constitute a different circuit at a differentinstance of time.

The above detailed description includes references to the accompanyingdrawings, which form a part of the detailed description. The drawingsshow, by way of illustration, specific examples that may be practiced.These examples are also referred to herein as “examples.” Such examplesmay include elements in addition to those shown or described. However,also contemplated are examples that include the elements shown ordescribed. Moreover, also contemplated are examples using anycombination or permutation of those elements shown or described (or oneor more aspects thereof), either with respect to a particular example(or one or more aspects thereof), or with respect to other examples (orone or more aspects thereof) shown or described herein.

In this document, the terms “a” or “an” are used, as is common in patentdocuments, to include one or more than one, independent of any otherinstances or usages of“at least one” or “one or more.” In this document,the term “or” is used to refer to a nonexclusive or, such that “A or B”includes “A but not B,” “B but not A,” and “A and B,” unless otherwiseindicated. In the appended claims, the terms “including” and “in which”are used as the plain-English equivalents of the respective terms“comprising” and “wherein.” Also, in the following claims, the terms“including” and “comprising” are open-ended, that is, a system, device,article, or process that includes elements in addition to those listedafter such a term in a claim are still deemed to fall within the scopeof that claim. Moreover, in the following claims, the terms “first,”“second,” and “third,” etc. are used merely as labels, and are notintended to suggest a numerical order for their objects.

The above description is intended to be illustrative, and notrestrictive. For example, the above-described examples (or one or moreaspects thereof) may be used in combination with others. Other examplesmay be used, such as by one of ordinary skill in the art upon reviewingthe above description. The Abstract is to allow the reader to quicklyascertain the nature of the technical disclosure. It is submitted withthe understanding that it will not be used to interpret or limit thescope or meaning of the claims. Also, in the above Detailed Description,various features may be grouped together to streamline the disclosure.However, the claims may not set forth every feature disclosed herein asexamples may feature a subset of said features. Further, examples mayinclude fewer features than those disclosed in a particular example.Thus, the following claims are hereby incorporated into the DetailedDescription, with a claim standing on its own as a separate example. Thescope of the examples disclosed herein is to be determined withreference to the appended claims, along with the full scope ofequivalents to which such claims are entitled.

What is claimed is:
 1. A system comprising at least one processor and amemory in communication with the at least one processor, wherein thesystem is programmed to: receive from a lead generator component firstexecution cycle data comprising a first set of leads and a second set ofleads, wherein the first set of leads comprises first lead informationdescribing a plurality of first potential customers and the second setof leads comprises second lead information describing a plurality ofsecond potential customers; determine by a lead scorer component a firstlead score for the plurality of first potential customers in the firstset of leads based at least in part on a first lead scoring model;determine that the first lead score for the first lead is greater than afirst lead score threshold for the plurality of first potentialcustomers in the first set of leads; determine by the lead scorercomponent that a second lead score for the plurality of second potentialcustomers in the second set of leads is less than the first lead scorethreshold; filter a set of filtered leads including the first leadinformation from the plurality of first potential customers in the firstset of leads; determine by an evaluator component that a quality of thefirst execution cycle data is different than a quality of analysiswindow data by more than a quality threshold, the analysis window datacomprising a plurality of leads received during an analysis window timeperiod; determine a second lead scoring model based at least in part onthe first execution cycle data; receive second execution cycle datacomprising a third set of leads comprising third lead informationdescribing a plurality of third potential customers; determine by thelead scorer component a third lead score for the plurality of thirdpotential customers in the third set of leads based at least in part onthe second lead scoring model wherein the lead scorer componentdetermines whether a guard-rail enforcer component controls thedetermined second lead scoring model; determine that the third leadscore for the plurality of third potential customers in the set of thirdleads is greater than the first lead score threshold; filter a secondset of filtered leads including the third lead information from thethird lead; and adjust at least one of the scoring models on the leadscorer component based on the quality of leads determined by theplurality of lead scoring models; wherein the determining that thequality of the first execution cycle data is different than a quality ofanalysis window data by more than a quality threshold comprises:determining a scrub rate, wherein the scrub rate describes a portion ofthe first execution cycle data having lead scores greater than the firstlead score threshold; determine that the scrub rate is greater than ananalysis window scrub rate by more than a scrub rate threshold; andwherein the system is further programmed to receive lead score datadescribing lead scores for at least one lead received during anobservation period and at least one lead received during the analysiswindow and not during the observation period.
 2. The system of claim 1,wherein the determining the second lead scoring model comprisesre-training the first lead score model based at least in part on thefirst execution cycle data.
 3. The system of claim 1, wherein thedetermining the second lead scoring model comprises modifying a scoringparameter of the first lead score model.
 4. The system of claim 1,further programmed to after determining the second lead scoring model,determine that application of the second lead scoring model is permittedby a limitation rule.
 5. The system of claim 1, further programmed to:determine an average scrub rate for a second plurality of leads receivedduring the analysis window; and determine the scrub rate threshold basedat least in part on the average scrub rate.
 6. The system of claim 1,further programmed to: determine a mean scrub rate of a plurality ofhistorical scrub rates to generate the analysis window scrub rate; anddetermine that the scrub rate threshold is more than two standarddeviations higher than the mean scrub rate.
 7. The system of claim 1,wherein determining that the quality of the first execution cycle datais different than a quality of analysis window data by more than aquality threshold comprises determining that a first value of the firstlead information indicates that the first lead is not likely to convert.8. The system of claim 1, where the system is further programmed to:before generating the second set of filtered leads, send a message to anadministrative user; and determine that more than a threshold time haspassed since the sending of the message.
 9. The system of claim 1,wherein the first execution cycle data consists of leads received duringan observation period shorter than the analysis window.
 10. The systemof claim 1, wherein the system is further programmed to determine ascrub rate for a plurality of lead scores for leads received during theanalysis window.
 11. The system of claim 1, wherein the first executioncycle data and the second execution cycle data consist of leads with afirst common value for a first lead category, and wherein the firstexecution cycle data and the second execution cycle data consist ofleads with a second common value for a second common lead value.
 12. Amethod for adaptively generating leads, comprising: receiving firstexecution cycle data comprising a first set of leads and a second set ofleads, wherein the first set of leads comprises first lead informationdescribing a first plurality of potential customers and the second setof leads comprises second lead information describing a second pluralityof potential customers; determining a first lead score for each of thefirst plurality of potential customers in the first set of leads basedat least in part on a first lead scoring model; determining that thefirst lead scores for the first set of leads is greater than a firstlead score threshold; determining that a second lead score for thesecond set of leads is less than the first lead score threshold;filtering a set of filtered leads including the first lead informationfrom the first set of leads; determining that a quality of the firstexecution cycle data is different than a quality of analysis window databy more than a quality threshold, the analysis window data comprising aplurality of leads received during an analysis window time period;determining a second lead scoring model based at least in part on thefirst execution cycle data; receiving second execution cycle datacomprising a third set of leads comprising third lead informationdescribing a plurality of third potential customers; determining a thirdlead score for each of the plurality of third potential customers in thethird set of leads based at least in part on the second lead scoringmodel including determining whether a guard-rail enforcer componentcontrols the determined second lead scoring model; determining that thethird lead score for the third lead is greater than the first lead scorethreshold; selecting a second set of filtered leads including the thirdlead information from the plurality of third leads; and adjust at leastone of the scoring models on the lead scorer component based on thequality of leads determined by the plurality of lead scoring models;wherein the determining that the quality of the first execution cycledata is different than a quality of analysis window data by more than aquality threshold comprises: determining a scrub rate, wherein the scrubrate describes a portion of the first execution cycle data having leadscores greater than the first lead score threshold; determine that thescrub rate is greater than an analysis window scrub rate by more than ascrub rate threshold; and receiving lead score data describing leadscores for at least one lead received during an observation period andat least one lead received during the analysis window and not during theobservation period.
 13. The method of claim 12, wherein the determiningthe second lead scoring model comprises re-training the first lead scoremodel based at least in part on the first execution cycle data.
 14. Themethod of claim 12, wherein the determining the second lead scoringmodel comprises modifying a scoring parameter of the first lead scoremodel.
 15. The method of claim 12, further comprising, after determiningthe second lead scoring model, determining that application of thesecond lead scoring model is permitted by a limitation rule.
 16. Themethod of claim 12, further comprising: determining an average scrubrate for a second plurality of leads received during the analysiswindow; and determining the scrub rate threshold based at least in parton the average scrub rate.
 17. The method of claim 12, furthercomprising: determining a mean scrub rate of a plurality of historicalscrub rates to generate the analysis window scrub rate; and determiningthat the scrub rate threshold is more than two standard deviationshigher than the mean scrub rate.
 18. A non-transitory machine-readablemedium comprising instructions which, when read by a machine, cause themachine to perform operations comprising: receiving first executioncycle data comprising a first set of leads and a second set of leads,wherein the first set of leads comprises first lead informationdescribing a first plurality of potential customers and the second setof leads comprises second lead information describing a second pluralityof potential customers; determining a first lead score for each of theplurality of potential customers in the set of first leads based atleast in part on a first lead scoring model; determining that the firstlead scores for the plurality of potential customers in the set of firstleads is greater than a first lead score threshold; determining that asecond lead score for the plurality of potential customers in the set ofsecond leads is less than the first lead score threshold; selecting aset of filtered leads including the first lead information from thefirst set of leads; determining that a quality of the first executioncycle data is different than a quality of analysis window data by morethan a quality threshold, the analysis window data comprising aplurality of leads received during an analysis window time period;determining a second lead scoring model based at least in part on thefirst execution cycle data; receiving second execution cycle datacomprising a third set of leads comprising third lead informationdescribing a plurality of third potential customers; determining a thirdlead score for each of the potential customer in the third set of leadsbased at least in part on the second lead scoring model includingdetermining whether a guard-rail enforcer component controls thedetermined second lead scoring model; determining that the third leadscores for the third leads are greater than the first lead scorethreshold; selecting a second set of filtered leads including the thirdlead information from the third set of leads; and adjust at least one ofthe scoring models on the lead scorer component based on the quality ofleads determined by the plurality of lead scoring models; wherein thedetermining that the quality of the first execution cycle data isdifferent than a quality of analysis window data by more than a qualitythreshold comprises: determining a scrub rate, wherein the scrub ratedescribes a portion of the first execution cycle data having lead scoresgreater than the first lead score threshold; determine that the scrubrate is greater than an analysis window scrub rate by more than a scrubrate threshold; and receiving lead score data describing lead scores forat least one lead received during an observation period and at least onelead received during the analysis window and not during the observationperiod.